On the elicitation of continuous, symmetric, 
unimodal distributions 



John Paul Gosling 
May 15, 2008 

Abstract 

In this brief note, we highlight some difficulties that can arise when 
fitting a continuous, symmetric, unimodal distribution to a set of expert's 
judgements. A simple analysis shows it is possible to fit a Cauchy distri- 
bution to an expert's beliefs when their beliefs actually follow a normal 
distribution. This example stresses the need for careful distribution fitting 
and for feedback to the expert about what the fitted distribution implies 
about their beliefs. 
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1 Elicitation 



In the context of statistical analysis, elicitation is the process of translating some- 
one's beliefs about some uncertain quantities into a probability distribution. An 
elicited probability distribution is often used as a prior distribution in a Bayesian 
analysis. In some application areas, there are few or no data av ailable and expert 
opinio n becomes paramoun t in t he decision making pro c ess. S ee lAng and Buttery 
(119971 1 , ICoolen et al.l (119921 ) and IChaloner and Rhamd (120011 ) for examples in nu- 
clear safety, engineering and clinical trials respectively. 

Elicitation is far from being a precise science. It can be difficult for experts 
to articulate their beliefs. There are also other complications due to the possible 
biases of the experts and the biases created by the questioning process. The 
process of questioning people about their beliefs is certainly not a new subject: 
it has been the focus of many psychological studies. Reviews of the psycho l ogica l 
liter ature from a sta t istical viewpoint can be found in lKadane and Wolfsonl (119981 ) 



and lO'Hagan et al.l (120061 ). In this note, we use a simple example to show the 



department of Probability and Statistics, 
j . p . gosling@sheffield .ac.uk 



University of Sheffield, Sheffield, UK, 



1 



implications to prior modelling of an expert's inability to articulate their beliefs 
with absolute precision. Throughout this note, we assume that the expert being 
questioned is female for ease of exposition and that there is a facilitator of the 
elicitation process conducting the elicitation exercise. 



2 Fitting distributions to percentile judgements 



The expert's beliefs for some continuous quantity X are to be modelled. Her 
density function for X is then f(X). The facilitator elicits information from her 
about /(.). She is not expected to accurately report the value of f(9) for all 
possible values of 9. In fact, she should not be e xpected to be able to report f(9) 



for any value of 9. IKadane and Wolfsonl (119981 ) suggest that quantiles or prob- 



abilities should be elicited, and, often, the expert is comfortable only providing 
the median and quartiles. 

For X, the expert is able to give the facilitator the following judgements: 
P(X < -0.6745) = 0.25, P{X < 0) = 0.5 and P(X < 0.6745) = 0.75; these 
are obviously the median and quartiles of the standard normal distribution. In 
addition to this, suppose that she is happy for the facilitator to assume that her 
density for X is unimodal. From the quartile judgements, a symmetric distribu- 
tion would be an appropriate fit to her judgements. There are infinitely many 
distributions that would satisfy these conditions. In Figure [TJ three possible 
cumulative distribution functions that interpolate her judgements precisely are 
plotted. The three distributions have widely varying tail-behaviours . 

In this example, we could try to discriminate between the possible fits by 
eliciting more judgements from the expert. Several methods have been pro- 
posed to get inform ation about the tails of an expert's distribution: tail ra- 
ti os are discussed in Kadane et al. (1980) and hypothetical data are discussed 
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Garthwaite and Al-Awadhil (120011 ). In these methods, further judgements are 



required fr om the expert in ord er to discover more about the tails of the distri- 
bution: in IKadane et al.l (11980), a judgement about the 90 th percentile are used 
to help characterise tail-behaviour. In our example, we suppose that the expert 
is able to specify P(X < -1.2816) = 0.1 and P(X < 1.2816) = 0.9 in addition 
to her earlier judgements. 

If we are uncertain as to whether the reported i th percentile is in fact the 
expert's (i — 2.5) th percentile or the (i + 2.5) th percentile, then we should take ac- 
count of this uncertainty. In fact, in many cases, the expert is onl y expected to be 



able to make probability judgements up to a precision of 0.05 (see lO'Hagan et al. 



20061 . for examples). In addition to this, we may suspect that she may only be 
able to give judgements about X to within ±0.05. This seems reasonable as 
this creates an interval that is a tenth of the her 'true' distribution's standard 
deviation. 

The graph in Figure [21 shows the results of fitting distributions to the expert's 
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Figure 1: The dots are the expert's judgements and the three lines are possible 
fits: normal (solid), t with 5 degrees of freedom (dotted) and Cauchy (dashed). 

judgements where independent uniform distributions are used to represent our 
uncertainty about the judgements. It is clear that we still could fit practically 
any unimodal symmetric distribution to her judgements. The true situation may 
be worse than what is depicted here as it is reasonable to expect the uncertainty 
to grow as she makes judgements further into the tails of her distribution. 



3 Summary 



Similar fitting procedures using percentile judgements are common in Bayesian 
analyses; normal di s tribut io ns were fitted to this type of exper t judgement in 



Cooke and Slijkhuisl ((20031) , iDenham and Mengersenl (120071 ) and Kennedy et al. 
(120081 ) . It is clear from the example of the previous section that the facilitator 



should check the sensitivity of their results to the possible tail-behaviours. Often, 
we suspect that the tail behaviour will make little difference; however, if there are 
no data or the data are far from what is expected a priori , then the tail-behaviour 
in th e elicited prior could have serious implications 
2006h . 
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Andrade and O'Hagan 



The models for eliciation described in lO'Hagan and Oakleyl (120041 ) and lGosling et al 



(120071 ) begin to address some of the uncertainties in the elicitation process; more 
specifically, the facilitator's uncertainty in the choice of distribution to fit. How- 
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Figure 2: The rectangles represent uniform uncertainty about the expert's five 
judgements and the three lines are possible fits: normal (solid), t with 5 degrees 
of freedom (dotted) and Cauchy (dashed). 

ever, uncertainty about the precision of the expert's judgements is not given full 
attention. One way to help prevent misrepresentation of an expert's beliefs is to 
include feedback stages in the elicitation exercise. During the feedback stage, the 
expert is shown the implications of using a particular distribution (for example, 
the tertiles of the fitted distribution) and asked if they wish to revise her original 
judgements and/or make additional judgements. Unfortunately, this additional 
procedure cannot eliminate the expert's inability to specify her judgement's with 
absolute precision. In conclusion, we reiterate that elicitation is not a precise 
science, and, in order to make progress, more effort should be made to develop 
elicitation models that accommodate all the forms of uncertainty that are clearly 
present in elicitation. 
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